Human machine interaction monitor

ABSTRACT

An apparatus comprises an interface and a control circuit. The interface may be configured to receive a plurality of sensor signals from a vehicle platform of a vehicle and present one or more control signals to the vehicle platform. The control circuit may be configured to (i) detect whether an attention state of a driver is in an attentive state or an inattentive state in response to one or more of the plurality of sensor signals from the vehicle platform during a first window having a first duration, (ii) assess whether the driver is sufficiently attentive by monitoring the one or more of the plurality of sensor signals from the vehicle platform and determining whether changes in the attention state of the driver during a second window having a second duration that is longer than the first duration exceeds a threshold, and (iii) when the threshold is exceeded, transition operation of the vehicle to the driver and safely discontinue an automation system function of the vehicle.

FIELD OF THE INVENTION

The invention relates to advanced driver assistance systems generallyand, more particularly, to a method and/or apparatus for implementing ahuman machine interaction monitor.

BACKGROUND

Production driver attention tracking features on the market today employalgorithms to track and categorize visual attention of drivers. However,driver attention tracking features cannot be relied upon solely to keepa driver functionally vigilant (and thereby meet safety goals) due tonumerous human factors related challenges in keeping the driver engagedin the driving task. Foreseeable misuse and types of abuse of similarfeatures that have been documented in the market today need to be takeninto account. The misuse can result in edge cases where a driverattention tracking feature labels the driver as fully aware therebykeeping an automation feature active while the driver cannot intervenein case of any hazardous event. Hence, there is a need to detect adriver who is no longer sufficiently attentive to act as a safety netfor the driver attention tracking feature.

It would be desirable to implement a human machine interaction monitorto ensure driver engagement during supervision of assisted(collaborative) driving automation.

SUMMARY

The invention concerns an apparatus comprising an interface and acontrol circuit. The interface may be configured to receive a pluralityof sensor signals from a vehicle platform of a vehicle and present oneor more control signals to the vehicle platform. The control circuit maybe configured to (i) detect whether an attention state of a driver is inan attentive state or an inattentive state in response to one or more ofthe plurality of sensor signals from the vehicle platform during a firstwindow having a first duration, (ii) assess whether the driver issufficiently attentive by monitoring the one or more of the plurality ofsensor signals from the vehicle platform and determining whether changesin the attention state of the driver during a second window having asecond duration that is longer than the first duration exceeds athreshold, and (iii) when the threshold is exceeded, transitionoperation of the vehicle to the driver and safely discontinue anautomation system function of the vehicle.

BRIEF DESCRIPTION OF THE FIGURES

Embodiments of the invention will be apparent from the followingdetailed description and the appended claims and drawings.

FIG. 1 is a diagram illustrating a system in accordance with anembodiment of the invention.

FIG. 2 is a diagram illustrating an example implementation utilizingglance behavior as a measure of driver attentiveness.

FIG. 3 is a flow diagram illustrating example operational states of asystem in accordance with an embodiment of the invention.

FIG. 4 is a diagram illustrating an implementation of an advanceddriver-assistance systems (ADAS) human machine interaction monitor inaccordance with an example embodiment of the present invention.

FIG. 5 is a diagram illustrating example criteria for acceptable andunacceptable driver attentiveness.

FIG. 6 is a diagram illustrating an example operation of a system inaccordance with an embodiment of the invention.

FIGS. 7-13 are diagrams illustrating example interactions between adriver and a system in accordance with an example embodiment of theinvention.

FIG. 14 is a diagram illustrating an electronic control unitimplementing an advanced driver-assistance systems (ADAS) featurecontrol system in accordance with an example embodiment of theinvention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention include providing a human machineinteraction monitor that may (i) provide a driver awareness levelescalation regime that generates warnings at a number (e.g., 3) ofdifferent levels via a human machine (or vehicle) interface (HMI) of avehicle, (ii) detect insufficient attention related to short term sharedvisual attention (e.g., from involvement in secondary tasks), (iii) takeinto account foreseeable misuse and types of abuse of similar featuresthat have been documented in the market, (iv) detect and mitigate edgecases where the driver is labeled as fully aware thereby keeping anautomation feature active while the driver is not able to intervene incase of a hazardous event, (v) provide additional functionality tomethodically detect a driver who is no longer sufficiently attentive toact as a safety net for a driver attention tracking feature, (vi) detectinsufficient detection of inattention when the HMI is not working orthere is driver misuse, (vii) track glance distribution over a longertime duration than a duration of time an awareness monitor tracksoff-road glance behavior, (viii) utilize outputs from both human machineinteraction monitoring and awareness monitoring to affect kinestheticsand longitudinal control of the vehicle, and/or (ix) be implemented asone or more integrated circuits.

A safety goal (e.g., Society of Automotive Engineers Level 2-3 (SAEL2+)) is generally imposed on partial (collaborative) driving automationfeatures to not be operational unless the driver is attentive. Anattentive driver can intervene in a timely manner to mitigate hazardousevents that can arise due to feature limitations. Driver inattention isof primary concern today with SAE L2+ automation systems that assumelateral and longitudinal steering functions. The safety case depends ona sufficiently attentive driver being able to take control in hazardoussituations where a limitation in the operational capability of theautomation system is reached. There is an ever growing need to designsafe and intelligent collaborative driving systems to provide thenecessary user value through automation while still conditioning theuser into appropriate attentional compliance. Hence, there is a need fora solution to methodically detect whether the driver is attentive andreach an appropriate safe state when the driver is no longer attentive.

In order to best achieve attentional compliance, a clear and consistentmental model for the driver needs to be established. Driverunderstanding of the capabilities and the limitations of an automationsystem may largely inform how the driver interacts with the automationsystem. An accurate representation of the mental model for the driver iscritical to (i) how likely drivers are to respond to safety criticalsituations where timely intervention is required, (ii) the developmentof trust in the system that can lead to over-reliance or under-use, and(iii) the overall concept the driver has of operation of the system.

Of especially complex concern is the extent to which semi-automatedsystems impact information processing capabilities and cognitiveoff-loading of the driver, including the likelihood of the driver toengage in distracting secondary tasks. As the role of the driver shiftsfrom full time active operator to intermittently passive supervisor,system design needs to ensure the driver is still able to perceivecritical changes in the driving environment and system status.

In various embodiments, a method and system are provided for monitoringdriver awareness, including providing feedback to the driver, evaluatingdriver behavior over time, and enabling advanced driver-assistancesystems (ADAS) functionality when driver awareness is at an acceptablelevel. In various embodiments, a driver awareness estimator (DAE) may beimplemented to monitor the attention level of the driver over time andprovide feedback (e.g., warnings) to the driver (e.g., using a humanmachine interface (HMI) of a vehicle). The HMI generally provides aconnection between the driver and the vehicle, such that reaction of thedriver to warnings may be observed, over an extended period of time, bythe driver awareness estimator system. In an example, a system inaccordance with an embodiment of the invention may utilize theobservations made using the DAE to improve the quality of driverattention assessment. In an example, various ADAS features may bedisabled when the driver behavior monitored over the extended period oftime exceeds a particular threshold. In various embodiments, the systemmay provide improved driver attention. The system may also reduce apossibility for a driver to override (e.g., cheat, game, etc.) a drivermonitoring system (DMS) of a vehicle.

In various embodiments, the DAE generally comprises two separatefunctionalities, an attention (or awareness) monitor and a human machineinteraction monitor (HMIM). The attention monitor may function todetermine a state of a driver (e.g., eyes on the road, inattentive,temporarily unaware, out-of-the-loop, dozing off, etc.) during short(e.g., a few seconds) windows of time. In an example, the attentionmonitor may track eye blinking or a line of sight of the driver using,for example, eye tracking information from a driver monitoring system ofthe vehicle, and then determines whether the driver is unaware or isinattentive. In an example, the attention monitor may perform eyetracking using a camera of the driver monitoring system. The attentionmonitor may also check whether a personal device or each device in thevehicle is used and then determine whether the driver is inattentive.The attention monitor may further determine the state of the driverbased on driving information, such as speed, steering angle, andvariability of the speed of the vehicle. In an example, the attentionmonitor may provide a driver awareness level escalation regime thatgenerates warnings at a number (e.g., 3, etc.) of different levels viathe human machine interface (HMI) and may detect insufficient attentionrelated to short term shared visual attention (e.g., from involvement insecondary tasks). The attention monitor may be implemented similarly toexisting attention tracking features on the market today that employalgorithms to track and categorize visual attention of drivers.

In various embodiments, the DAE does not rely solely on the attentionmonitor to keep the driver functionally vigilant (and thereby meet thesafety goal) due to numerous human factors related challenges in keepingthe driver engaged in the driving task. In various embodiments, the DAEmay account for foreseeable misuse and the types of abuse of similarfeatures that have been documented in the market today. For example,misuse may result in edge cases where the attention monitor labels thedriver as fully aware, thereby keeping an automation feature active,while the driver cannot intervene in case of any hazardous event. Hence,the DAE in accordance with an embodiment of the invention generallyincludes an additional functionality to methodically detect a driver whois no longer sufficiently attentive to act as a safety net for theattention monitor. The additional functionality is generally provided bythe human machine interaction monitor (HMIM) in accordance with anembodiment of the invention. The HMIM generally provides a morediversified assessment of driver inattention, providing an additionallayer of safety on top of the attention monitor.

Referring to FIG. 1 , a diagram is shown illustrating a system inaccordance with an embodiment of the invention. In an example, a system(or apparatus) 90 may implement an advanced driver-assistance system(ADAS). In various embodiments, the system 90 may comprise a vehicleplatform 92, a driver monitoring system (DMS) 94, a human machine (orvehicle) interface (HMI) 96, and a function control module 100. Invarious embodiments, the vehicle platform 92, the driver monitoringsystem (DMS) 94, and the function control module 100 may be implementedas Automotive Safety Integrity Level (ASIL), while the human machineinterface (HMI) 96 may be implemented as Quality Management (QM).

Automotive Safety Integrity Level (ASIL) is a risk classification schemedefined by the ISO 26262—Functional Safety for Road Vehicles standard.This is an adaptation of the Safety Integrity Level (SIL) used in IEC61508 for the automotive industry. The ASIL classification helpsdefining the safety requirements needed to be in line with the ISO 26262standard to keep the risk to an acceptable level. The ASIL isestablished by performing a risk analysis of a potentially hazardousscenario by looking at the Severity, Exposure and Controllability of thevehicle operating scenario. The safety goal for that hazardous scenarioin turn carries the ASIL requirements. The ASILs range from ASIL D,representing the highest degree of risk of a hazardous scenario turninginto a mishap and highest degree of rigor needed to be applied in theassurance of the resultant safety requirements, to QM, representingapplications with no automotive hazardous scenarios with unacceptablerisk and, therefore, no safety requirements to manage under the ISO26262 safety processes. The level QM, referring to “Quality Management”,means that risk associated with a hazardous event is not unreasonableand does not therefore require safety measures in accordance with ISO26262. The intervening levels (ASIL C, ASIL B, and ASIL A) are simply arange of varying degrees of hazard risk levels and degrees of assurancerequired.

The standard defines functional safety as “the absence of unreasonablerisk due to hazards caused by malfunctioning behavior of electrical orelectronic systems.” ASILs establish safety requirements, based on theprobability and severity of harm, for automotive components to becompliant with ISO 26262. Systems like airbags, anti-lock brakes, andpower steering require an ASIL D grade-the highest rigor applied tosafety assurance-because the risks associated with their failure are thehighest. On the other end of the safety spectrum, components like wipersystems require only an ASIL A grade. Headlights and brake lightsgenerally would be ASIL B, as would be rear lights due to risk of rearcollisions, while automatic emergency brake systems would generally beASIL C due to risks associated with the unintended deceleration.

In an example, the vehicle platform 92, the DMS 94, and the HMI 96 mayprovide input signals to the function control module 100. In an example,the vehicle platform 92 may provide an input signal (e.g., VEHICLESPEED) communicating vehicle speed to the function control module 100.The DMS 94 may provide input signals communicating information relatedto driver awareness (e.g., driver eye movement, driver hand positions,steering angle, etc.). In an example, the HMI 96 may provide a firstinput signal (e.g., ACTIVATION REQUEST) and a second input signal (e.g.,DEACTIVATION REQUEST) to the function control module 100. The signalACTIVATION REQUEST may communicate a request from the driver to activatean ADAS feature controlled by the function control module 100. Thesignal DEACTIVATION REQUEST may communicate a request from the driver tode-activate the ADAS feature controlled by the function control module100. In some embodiments, the HMI 96 may optionally present an inputsignal (e.g., DRIVER INFO) communicating information about theparticular driver operating the vehicle. In various embodiments, thesignals VEHICLE SPEED and DECELERATION REQUEST may be implemented asASIL, and the signals ACTIVATION REQUEST and DEACTIVATION REQUEST may beimplemented as QM.

In an example, the function control module 100 may provide outputsignals to the vehicle platform 92 and the HMI 96. In an example, thefunction control module 100 may present an output signal (e.g.,DECELERATION REQUEST) to the vehicle platform 92. The signalDECELERATION REQUEST may be configured to allow the function controlmodule 100 to bring the vehicle to a safe stop. The function controlmodule 100 may present a signal (e.g., DRIVER WARNING) to the HMI 96.The signal DRIVER WARNING may communication information to cause the HMI96 to present a particular warning to the driver. In variousembodiments, the signal DRIVER WARNING may be implemented as QM.

In an example, the function control module 100 may comprise a block (orcircuit) 102, a block (or circuit) 104, and a block (or circuit) 106.The block 102 may be implemented as an attention (or awareness) monitor.The block 104 may be implemented as a human machine interaction monitor(HMIM). The block 106 may be implemented as ADAS feature mode manager.In one example, the block 106 may be implemented as an autopilot modemanager. In various embodiments, the blocks 102, 104, and 106 aregenerally implemented as ASIL. In an example, the signal VEHICLE SPEEDmay be presented to a first input of the block 102, a first input of theblock 104, and a first input of the block 106. The signals from the DMS94 may be presented to a second input of the block 102 and a secondinput of the block 104. The block 102 may present a signal (e.g.,AWARENESS LEVEL) to a third input of the block 104 and a second input ofthe block 106. The signal AWARENESS LEVEL may be implemented as ASIL.The block 104 may present a signal (e.g., SUFFICIENTLY ATTENTIVE) to athird input of the block 106. The signal SUFFICIENTLY ATTENTIVE may beimplemented as ASIL. In embodiments where the HMI 96 provides the signalDRIVER INFO to the function control module 100, the signal DRIVER INFOmay be presented to a fourth input of the block 104.

In various embodiments, the block 102 and the block 104 may beconfigured as a driver awareness estimator (DAE) to methodically detectwhether a driver is attentive and reach an appropriate safe state whenthe driver is no longer attentive. In an example, the attention monitor102 may provide a driver awareness level escalation regime thatgenerates warnings at a number (e.g., 3) of different levels via the HMI96 and may detect insufficient attention related to short term sharedvisual attention (e.g., from involvement in secondary tasks). Theattention monitor 102 may be implemented similarly to existingproduction attention tracking features on the market today that employalgorithms to track and categorize visual attention of drivers.

In various embodiments, the driver awareness estimator (DAE) does notrely solely on the attention monitor 102 to keep the driver functionallyvigilant (and thereby meet the safety goal) due to numerous humanfactors related challenges in keeping the driver engaged in the drivingtask. In various embodiments, the driver awareness estimator (DAE) mayaccount for foreseeable misuse and the types of abuse of similarfeatures that have been documented in the market today. For example,misuse can result in edge cases where the attention monitor 102 labelsthe driver as fully aware thereby keeping an automation featurecontrolled by the function control module 100 active while the drivercannot intervene in case of any hazardous event. Hence, the driverawareness estimator (DAE) in accordance with an embodiment of theinvention generally utilizes the HMIM 104 to provide an additionalfunctionality to methodically detect a driver who is no longersufficiently attentive to act as a safety net for the attention monitor102.

The HMIM 104 is generally configured to detect insufficient attentionwhen the HMI 96 is not working or there is driver misuse. In an example,the HMIM 104 may look at the off-road glance distribution pattern of thedriver by analyzing the toggle behavior between awareness levelsreported by attention monitor 102 over a longer duration of time thanthe duration (or window) of time used by the attention monitor 102. Invarious embodiments, long-term glance distribution patterns may be usedto affect the kinesthetics and longitudinal control of the vehicleplatform 92. In an example, the HMIM 104 may focus on a longer-termassessment based on toggle behavior between attentiveness statesreported by the attention monitor 102. By monitoring the awareness levelof the driver (e.g., as captured by the time-distribution of theawareness states) in a given time window, a tunable (or programmable)number of transitions and an acceptable total time within each state ofawareness may be defined. Using assisted driving (e.g., adaptive cruisecontrol (ACC), etc.) glance behavior as the gold standard, driverengagement may be calculated based on the glance distribution patterns.The HMIM 104 generally prevents the driver from repeatedly entering intolower awareness states over a longer duration of time that may affectdriver controllability by assessing the longer-term glance patterns ofthe driver and then triggering a transfer of control to the driver and atransition of the vehicle to a safe state (e.g., via the signalDECELERATION REQUEST, etc.).

Referring to FIG. 2 , a diagram is shown illustrating an exampleimplementation of a driver awareness estimator utilizing glance behavioras a measure of driver attentiveness. Design of increasingly advancedsupervised systems is complex due to the critical role of thehuman-in-the-loop. Successful design ensures the driver is engaged andable to take over when needed during alerted failures, silent failures,and other transitions of control. Developing a forced vigilance systeminvolves a deep understanding of user perception, cognition, andresponse behavior. In an example, the attention monitor 102 generallyimplements a driver awareness level escalation regime that may generatewarnings at a number of different levels via the HMI 96 and detectsinsufficient attention related to short term shared visual attention(e.g., from involvement in a secondary tasks). In an example, theattention monitor 102 may observe driver glance behavior 108 over ashort duration of time window 110 to determine an awareness level. In anexample, if over the short period of time (e.g., a few seconds) thedriver is eyes-off-road 50% of the time or over a longer period of time(e.g., 4-5 times as long as the short period) the driver iseyes-off-road 30% of the time, the attention monitor 102 may indicatethe driver is not aware. In an example, the attention monitor 102 maygenerate warnings at three different levels: temporarily unaware,unaware, out of the loop. In an example, the number of warnings mayinclude, but are not limited to, auditory and visual reminders, hapticreminder (e.g., seat vibration), hands on, reduced propulsion, take overrequest, and slow into a safe stop.

In an example, the HMIM 104 may observe the driver glance behavior 108over a long duration of time window 112 to determine whether the driveris sufficiently aware. In an example, the HMIM 104 may detectinsufficient attention when the HMI 96 is not working as desired (e.g.,given that the HMI 96 is typically QM) or there is misuse, by looking atthe off-road glance distribution over a longer period of time than theattention monitor 102. In an example, the HMIM 104 may be configured toutilize the HMI 96 and the attention monitor 102 to detect whether theHMI 96 is successfully sending signals to the driver. For example,hardware or tracking failure generally means the driver is not receivingsignals from the HMI 96. In another example, the HMIM 104 may beconfigured to utilize the HMI 96 and the attention monitor 102 to detectwhether the driver is gaming (abusing) the system 90. For example, thedriver may be misusing the system by maximizing eyes-off-road timecontinuously by bouncing between aware and temporarily unaware states.

Referring to FIG. 3 , a flow diagram is shown illustrating exampleoperational states of the HMIM system in accordance with an embodimentof the invention. In an example, a feature control process 200 maycomprise a plurality of states of the HMIM system in accordance with anembodiment of the invention. In an example, the plurality of states maycomprise a number of feature states and a number of driver awarenessstates. In various embodiments, the HMIM 104 monitors the awarenesslevel of the driver (e.g., as captured by a time-distribution of thedriver awareness states of the driver awareness estimator (DAE)) anddefines an acceptable number of transitions and/or an acceptable totaltime within a particular DAE state. In an example, the control process(or method) 200 may comprise a step (or state) 202, a step (or state)204, a step (or state) 206, a step (or state) 208, a step (or state)210, a step (or state) 212, a step (or state) 214, a step (or state)216, a step (or state) 218, a step (or state) 220, a step (or state)222, a step (or state) 224, a step (or state) 226, a step (or state)228, a step (or state) 230, a step (or state) 232, a step (or state)234, and a step (or state) 236.

In an example, the process 200 may begin in the state 202 with the ADASfeature OFF and transition to the state 204. In the state 204, the ADASfeature is not ready for activation. The ADAS feature may remain notready for activation until operational design domain (ODD) conditions210 are appropriate for activation. The operational design domain (ODD)safety concept ensures a Society of Automotive Engineers Level 2-3 (SAEL2+) driver assistance feature is acceptably safe by reducing theexposure to challenging operational situations. Challenging operationalsituations are operational situations judged to be outside the knowncapabilities of advanced driver-assistance systems (ADAS) and,therefore, are considered hazardous. The goal of the ODD safety conceptis to be able to identify at least 99% of operational situations inorder to minimize exposure to hazard scenarios.

When the ODD conditions 210 are appropriate for activation, the process200 may move to the state 206. In the state 206, the ADAS feature isready for activation. The process 200 may remain in the state 206 untilthe driver is observed to be in the state 212 (e.g., hands on steeringwheel and eyes on road). When the driver is in the state 212, theprocess 200 may move to the state 208, upon receiving a driveractivation request 214. In the state 208, the ADAS feature is active.With the ADAS feature active, the process 200 may monitor the awarenesslevel of the driver as captured by the time-distribution of DAE states220-226.

The process 200 may define acceptable numbers of awareness statetransitions and/or an acceptable total time within a particular DAEstate. In an example, the process 200 may set a particular duration(e.g., N minutes) and numbers of transitions (e.g., I, J, and K) fromthe driver awareness state 220, where the driver is considered awarebased on an eyes on road observation 230, to the driver awareness states222, 224, and 226, respectively. In an example, the process 200 may seta first criterion (or threshold) 232 for determining whether the driveris considered to be in a temporarily unaware state 222 (e.g., the numberof times the driver enters the state 222 is greater than or equal to I),a second criterion (or threshold) 234 for determining whether the driveris considered to be in an unaware state 224 (e.g., the number of timesthe driver enters the state 224 is greater than or equal to J), and athird criterion (or threshold) 236 for determining whether the driver isconsidered to be in an out-of-the-loop state 226 (e.g., the number oftimes the driver enters the state 226 is greater than or equal to K). Inan example, the values I, J, and K may represent a maximum acceptablenumber of transitions of the respective driver awareness state duringthe particular duration selected. In an example, the process 200 maymove to the state 240 when one or more of the criteria 232, 234, and 236are met during the particular duration selected. In an example, theduration N and the criteria 232, 234, and 236 may be programmable. In anexample, the thresholds I, J, and K may be similar or different. In anexample, the thresholds I, J, and K may be set based on a profile of aparticular driver. In an example, the criteria 232, 234, and 236 may bemodified as the HMIM 104 learns a behavior (e.g., glance behavior)distribution of a particular driver.

In the state 240, the process 200 may notify the driver to take overoperation of the vehicle and reduce the vehicle propulsion to aparticular speed that is considered safe (e.g., 5 kph). The process 200may then move to the state 204, where the ADAS feature remains not readyfor activation until the driver takes a particular action (e.g., cyclesthe ignition switch).

Referring to FIG. 4 , a diagram is shown illustrating an implementationof the system 100 in accordance with an example embodiment of theinvention. In an example, the apparatus (or system) 100 may be mountedtotally within, or at least partially within a vehicle 50. In anexample, the system (or apparatus) 100 may be implemented as part of anadvanced driver-assistance systems (ADAS) electronic control unit (ECU)90. In various embodiments, the system 100 implementing the driverattentiveness estimator (DAE) may be implemented within the ADAS ECU 90of the vehicle 50. The ADAS ECU 90 may be connected to the vehicleplatform 92 of the vehicle 50. The vehicle 50 may include the drivermonitoring system (DMS) 94, the human machine interface (HMI) 96, aforward looking camera (FLC) 250, a number of corner radar sensors 252a-252 d, a number of front side radar sensors (not shown), a forwardlooking radar (FLR) sensor 254, a high-definition (HD) map receiver 260,a global navigation satellite system (GNSS) receiver 262, and aninertial measurement unit (IMU) 264. In some embodiments, the vehicle 50may also include LIDAR sensors and/or sonar sensors (not shown).

The forward looking camera (FLC) 250 is generally used to detect andidentify objects and road features in front of the vehicle 50. In anexample, the forward looking camera (FLC) 250 may be configured toprovide stereoscopic vision with a 100-degree field of view (FOV). In anexample, the forward looking camera (FLC) 250 may be used to detect roadmarkings (e.g., lane markings, etc.), road signs, traffic lights,structures, etc. The corner radar sensors 252 a-252 d and the forwardlooking radar (FLR) sensor 254 (and LIDAR and/or sonar sensors whenpresent) are generally used to detect and track objects. In an example,each of the corner radar sensors 252 a-252 d may have a 140-degree FOV.In an example, the forward looking radar sensor (FLR) 254 may have twoFOVs, an 18-degree FOV for long-range sensing and a 90-degree FOV forshort range sensing. The IMU 264 generally reports the orientation,angular velocity and acceleration, and forces acting on the vehicle 50.

In an example, the DMS 94, the HD map receiver 260, the GNSS receiver262, the FLC 250, the FCRs 252 a-252 b, and the FLR 254 may be connectedto the system 90. In an example, the DMS 94, the HD map receiver 260,the GNSS receiver 262, the FLC 250, the FCRs 252 a-252 b, and the FLR254 may be connected to the system 90 via one or more vehicle buses ofthe vehicle 50. In another example, the DMS 94, the HD map receiver 260,the GNSS receiver 262, the FLC 250, the FCRs 252 a-252 b, and the FLR254 may be connected to the system 90 via a wireless protocol. In anexample, the DMS 94 may convey driver attentiveness information to thesystem 90. The FLC 250 may convey surrounding road information (e.g.,lane widths, marker types, lane marker crossing indications, and video)to the system 90. The GNSS receiver 262 may convey position data (e.g.,latitude value, longitude value, adjustment information and confidenceinformation) to the system 90. The HD map receiver 260 may transfer mapdata to the system 90.

The FLC 250 may implement an optical sensor. In various embodiments, theFLC 250 may be an optical camera. The FLC 250 is generally operationalto provide the surrounding road information (or image data) to thesystem 90. The road information may include, but is not limited to, lanewidth data, marker type data, lane change indicators, and video of aroadway ahead of the vehicle 50 within the field of view of the FLC 250.In various embodiments, the FLC 250 may be a color camera. The color maybe useful for distinguishing between solid-yellow lane markers (e.g.,leftmost lane markers) from solid-white lane markers (e.g., rightmostlane markers). In various embodiments, the FLC 250 may provide anestimated lane width for at least a current lane in the center of thefield of view of the FLC 250. In some embodiments, the FLC 250 mayprovide estimated lane widths for the lane(s) neighboring the centerlane. In other embodiments, the FLC 250 may provide estimated lanewidths for all of the lanes within the field of view of the FLC 250. Thelane widths may be determined using standard image recognition methodsand standard analysis methods implemented in the FLC 250. The FLC 250may also identify all lane markers within the field of view of the FLC250. When the FLC 250 crosses over a lane marker, the FLC 250 may notifythe system 90 that a lane change is occurring. Identification of thelane markers and the lane changes may be determined using standard imagerecognition methods and standard analysis methods implemented in the FLC250. The FLC 250 may transfer the road information to the system 90 viaa vehicle bus or a wireless protocol.

One or more other types of sensors may be used in conjunction with theFLC 250. Example sensors may include, but are not limited to, radarsensors, light detection and ranging (LiDAR) sensors, inertial sensors,thermal imaging sensors, and/or acoustic sensors. Some of the sensorsmay detect objects on the side of the road to provide estimations of aleft boundary and a right boundary of the road. From the left boundaryand the right boundary, a width of the road may be calculated. From thecalculated width, an estimation of how many lanes probably fit withinthe width may be made based on a standard lane width. Thereafter, thesensors may estimate the current lane that the vehicle 50 occupies basedon the relative distances of the sensors on the vehicle 50 to the leftboundary and the right boundary of the road and the estimated number oflanes. Lane crossovers may be determined by the sensors based on theestimated numbers of lanes and changes in the relative distances to theleft boundary and/or the right boundary.

The system 90 may implement a control circuit (e.g., an electroniccontrol unit). The system 90 is generally operational to keep track ofthe current lane that the vehicle 50 occupies and correct the currentposition of the vehicle 50 to a center of the current lane. The trackingmay be based on the satellite position data received in the GNSSreceiver 262, the map data received from the HD map receiver 260, andthe road information received in the vision detections from the FLC 250and the radar detections received from the FCRs 252 a-252 b and the FLR254. The satellite position data may include an adjustment value and acorresponding confidence value.

The GNSS receiver 262 may implement a satellite-navigation device. Invarious embodiments, the GNSS receiver 262 may include a GlobalPositioning System (GPS) receiver. Other types of satellite-navigationdevices may be implemented to meet the design criteria of a particularapplication. The GNSS receiver 262 is generally operational to providethe latitude data and the longitude data of the vehicle 50 based on theGNSS signals received from a number of satellites. The GNSS receiver 262may also be operational to adjust the latitude data and the longitudedata based on the adjustment value and a corresponding confidence valuereceived from the system 90. The confidence value may have a range fromzero (e.g., unreliable) to one (e.g., reliable). If the confidence valueis above a high threshold (e.g., >0.7), the GNSS receiver 262 maycorrect the latitude data and the longitude data per the adjustmentvalue. If the confidence value is below a low threshold (e.g., <0.3),the GNSS receiver 262 may ignore the adjustment value. If the confidencevalue is between the high threshold and the low threshold, the GNSSreceiver 262 may apply a correction to both the latitude data and thelongitude data that is a linear weighting based on the degree ofconfidence.

The HD map receiver 260 may implement a radio-frequency receiver. The HDmap receiver 260 may be operational to receive the map data from anantenna (not shown). The map data may be converted to a digital form andpresented to the system 90.

Referring to FIG. 5 , a diagram is shown illustrating example criteriafor acceptable and unacceptable driver attentiveness. In an example, agraph 300 illustrates a curve 302 representing an acceptable driverawareness distribution and a curve 304 representing an unacceptabledriver attention distribution. In an example, the curve 302 generallyrepresents a glance distribution that provides a desired controllabilityfor a particular population. The curve 304 generally represents a glancedistribution that does not provide the desired controllability for theparticular population.

In an example, the attention monitor 102 generally implements a driverawareness level escalation regime that may generate warnings at a numberof different levels (e.g., 306, 308, and 310) via the HMI 96 and detectsinsufficient attention related to short term shared visual attention(e.g., from involvement in a secondary tasks). In an example, theattention monitor 102 may observe driver glance behavior over a shortduration of time to determine an awareness level. In an example, if overa short period of time (e.g., a few seconds) the driver is eyes-off-road50% of the time or over a longer period of time (e.g., 4-5 times as longas the short period) the driver is eyes-off-road 30% of the time, theattention monitor 102 may indicate the driver is not aware. In anexample, the attention monitor 102 may generate warnings at threedifferent levels: temporarily unaware, unaware, out of the loop.However, other numbers of levels may be implemented to meet designcriteria of a particular application.

In an example, the number of warnings may include, but are not limitedto, auditory and visual reminders, haptic reminder (e.g., seatvibration), hands on, reduced propulsion, take over request, and slowinto a safe stop. In an example, for the temporarily unaware level, theattention monitor 102 may generate the warning 306 comprising auditoryand visual reminders. For the unaware level, the attention monitor 102may generate the warning 308 comprising auditory and visual remindersplus seat vibration, hands on and reduced propulsion. For theout-of-the-loop level, the attention monitor 102 may generate thewarning 310 comprising auditory and visual reminders, seat vibration,hands on, reduced propulsion, plus a take over request and slowing thevehicle to a safe stop.

In an example, the HMIM 104 generally tracks the output of attentionmonitor 102, which generally provides a level of inattention for thedriver as shown on the x-axis. If at any time, the glance distributionchanges from the glance distribution that provides a desiredcontrollability for a particular population (e.g., the curve 302) to aglance distribution that does not provide the desired controllabilityfor the particular population (e.g., the curve 304), the HMIM 104 mayassert control and indicate that the driver is “insufficientlyattentive” even if the attention monitor 102 at the instant says thedriver is aware. The HMIM 104 and the attention monitor 102 generallyoperate in different time horizons.

Referring to FIG. 6 , a diagram is shown illustrating an exampleoperation of an HMIM system in accordance with an embodiment of theinvention. In an example, the attention monitor 102 may be configured topresent a signal having a first state indicating the eyes of the driverare on the road and a second state indicating the eyes of the driver arenot on the road. In an example, the HMIM 104 may be configured tocapture and increment (or count or accumulate) states of inattention(e.g., temporarily unaware, unaware, and out of loop) as the states ofinattention are output by the attention monitor 102 during a particularperiod of time (e.g., 15 minutes). In the example illustrated, thedriver is bouncing between the aware state and the temporarily unawarestate. Each time the driver enters the temporarily unaware state theHMIM 104 logs the incident of inattention (e.g., by incrementing a countof the number of transitions for each inattentive state). Once the HMIM104 captures (accumulates) six incidents (transitions) within the 15minute time window, the HMIM 104 transitions control back to the driverand sends the vehicle into a safe stop mode. In an example, a similarconsequence may also be triggered by the driver entering into theunaware state three times or the out-of-the-loop state one time withinthe 15 minute time window.

Referring to FIGS. 7-13 , diagrams are shown illustrating exampleinteractions between a driver and a driver attentiveness estimationsystem in accordance with an embodiment of the invention. In an example,the HMIM 104 may be configured as described above in connection withFIG. 6 . In an example, the HMIM 104 may be configured to monitor theoutput by the attention monitor 102 using a period (monitoring duration)of fifteen minutes. In an example, the HMIM 104 may be configured totransition control back to the driver and send the vehicle into a safestop mode in response to capturing six incidents of the driver beingtemporarily unaware within the 15 minute time window. A similarconsequence may also be triggered by the driver entering into theunaware state three times or the out-of-the-loop state one time withinthe 15 minute time window.

Referring to FIG. 7 , a diagram is shown illustrating the HMIM 104capturing a first incident of the driver being temporarily unawarewithin the 15 minute time window. A picture 500 a and a picture 500 bare shown illustrating a view within a cockpit of a vehicle from theperspective of the driver. A display 502 a illustrates a current statusfor the criteria used by the HMIM 104 to determine whether to transitioncontrol back to the driver and send the vehicle into a safe stop modeprior to any incidents of inattentiveness. A circle 504 a is shownindicating where the attention of the driver is directed. In an example,the driver is driving hands-free with an ADAS feature active on thehighway. The attention of the driver is toward the forward roadway. Theattention monitor 102 has not reported any transitions from the awarestate to the temporarily unaware, unaware, or out-of-loop states.

In the picture 500 b, a display 502 b illustrates the current status forthe criteria used by the HMIM 104 to determine whether to transitioncontrol back to the driver and send the vehicle into a safe stop modeafter a first incident of the driver being labeled as temporarilyunaware by the attention monitor 102 at four minutes into the 15 minutetime window. A circle 504 b is shown indicating where the attention ofthe driver is directed when the attention monitor 102 labeled the driveras temporarily unaware. In an example, the driver receives a textmessage and the attention of the driver moves from the forward roadwayto the phone. The HMI 96 alerts the driver to pay attention to the road.The HMIM 104 increments the temporarily unaware criteria one incident.

Referring to FIG. 8 , a diagram is shown illustrating the HMIM 104capturing a second incident of the driver being temporarily unawarewithin the 15 minute time window. A picture 500 c and a picture 500 dare shown illustrating the view within the cockpit of the vehicle fromthe perspective of the driver. A display 502 c illustrates a currentstatus for the criteria used by the HMIM 104 to determine whether totransition control back to the driver and send the vehicle into a safestop mode following the first incident of inattentiveness at fourminutes into the 15 minute window. A circle 504 c is shown indicatingwhere the attention of the driver is directed. In an example, the drivercomplied to the attention request. The driver is driving hands-free withthe ADAS feature remaining active on the highway. The attention of thedriver is toward the forward roadway.

In the picture 500 d, a display 502 d illustrates the current status forthe criteria used by the HMIM 104 to determine whether to transitioncontrol back to the driver and send the vehicle into a safe stop modeafter the second incident of the driver being labeled as temporarilyunaware by the attention monitor 102 at six minutes into the 15 minutetime window. A circle 504 d is shown indicating where the attention ofthe driver is directed when the attention monitor 102 labeled the driveras temporarily unaware. In an example, the driver receives another textmessage and the attention of the driver moves from the forward roadwayto the phone. The HMI 96 alerts the driver to pay attention to the road.The HMIM 104 increments the temporarily unaware criteria one incident totwo incidents total.

Referring to FIG. 9 , a diagram is shown illustrating the HMIM 104capturing a third incident of the driver being temporarily unawarewithin the 15 minute time window. A picture 500 e and a picture 500 fare shown illustrating the view within the cockpit of the vehicle fromthe perspective of the driver. A display 502 e illustrates a currentstatus for the criteria used by the HMIM 104 to determine whether totransition control back to the driver and send the vehicle into a safestop mode following the second incident of inattentiveness at sixminutes into the 15 minute window. A circle 504 e is shown indicatingwhere the attention of the driver is directed. In an example, the drivercomplied to the attention request. The driver is driving hands-free withthe ADAS feature remaining active on the highway. The attention of thedriver is again toward the forward roadway.

In the picture 500 f, a display 502 f illustrates the current status forthe criteria used by the HMIM 104 to determine whether to transitioncontrol back to the driver and send the vehicle into a safe stop modeafter the third incident of the driver being labeled as temporarilyunaware by the attention monitor 102 at ten minutes into the 15 minutetime window. A circle 504 f is shown indicating where the attention ofthe driver is directed when the attention monitor 102 labeled the driveras temporarily unaware. In an example, the driver becomes distractedlooking for an item in the glove compartment of the vehicle. The HMI 96alerts the driver to pay attention to the road. The HMIM 104 incrementsthe temporarily unaware criteria one incident to three incidents total.

Referring to FIG. 10 , a diagram is shown illustrating the HMIM 104capturing a fourth incident of the driver being temporarily unawarewithin the 15 minute time window. A picture 500 g and a picture 500 hare shown illustrating the view within the cockpit of the vehicle fromthe perspective of the driver. A display 502 g illustrates a currentstatus for the criteria used by the HMIM 104 to determine whether totransition control back to the driver and send the vehicle into a safestop mode following the third incident of inattentiveness at ten minutesinto the 15 minute window. A circle 504 g is shown indicating where theattention of the driver is directed. In an example, the driver compliedto the attention request. The driver is driving hands-free with the ADASfeature remaining active on the highway. The attention of the driver isagain toward the forward roadway.

In the picture 500 h, a display 502 h illustrates the current status forthe criteria used by the HMIM 104 to determine whether to transitioncontrol back to the driver and send the vehicle into a safe stop modeafter the fourth incident of the driver being labeled as temporarilyunaware by the attention monitor 102 at twelve minutes into the 15minute time window. A circle 504 h is shown indicating where theattention of the driver is directed when the attention monitor 102labeled the driver as temporarily unaware. In an example, the driverbecomes distracted looking for an item in the glove compartment of thevehicle. The HMI 96 alerts the driver to pay attention to the road. TheHMIM 104 increments the temporarily unaware criteria one incident tofour incidents total.

Referring to FIG. 11 , a diagram is shown illustrating the HMIM 104capturing a fifth incident of the driver being temporarily unawarewithin the 15 minute time window. A picture 500 i and a picture 500 jare shown illustrating the view within the cockpit of the vehicle fromthe perspective of the driver. A display 502 i illustrates a currentstatus for the criteria used by the HMIM 104 to determine whether totransition control back to the driver and send the vehicle into a safestop mode following the fourth incident of inattentiveness at twelveminutes into the 15 minute window. A circle 504 i is shown indicatingwhere the attention of the driver is directed. In an example, the drivercomplied to the attention request. The driver is driving hands-free withthe ADAS feature remaining active on the highway. The attention of thedriver is again toward the forward roadway.

In the picture 500 j, a display 502 j illustrates the current status forthe criteria used by the HMIM 104 to determine whether to transitioncontrol back to the driver and send the vehicle into a safe stop modeafter the fifth incident of the driver being labeled as temporarilyunaware by the attention monitor 102 at thirteen minutes into the 15minute time window. A circle 504 j is shown indicating where theattention of the driver is directed when the attention monitor labeledthe driver as temporarily unaware. In an example, the driver becomesdistracted looking out a window of the vehicle at scenery passing by thevehicle. The HMI 96 alerts the driver to pay attention to the road. TheHMIM 104 increments the temporarily unaware criteria one incident tofive incidents total.

Referring to FIG. 12 , a diagram is shown illustrating the HMIM 104capturing a sixth incident of the driver being temporarily unawarewithin the 15 minute time window. A picture 500 k and a picture 500 lare shown illustrating the view within the cockpit of the vehicle fromthe perspective of the driver. A display 502 k illustrates a currentstatus for the criteria used by the HMIM 104 to determine whether totransition control back to the driver and send the vehicle into a safestop mode following the fifth incident of inattentiveness at thirteenminutes into the 15 minute window. A circle 504 k is shown indicatingwhere the attention of the driver is directed. In an example, the drivercomplied to the attention request. The driver is driving hands-free withthe ADAS feature remaining active on the highway. The attention of thedriver is toward the forward roadway.

In the picture 500 l, a display 502 l illustrates the current status forthe criteria used by the HMIM 104 to determine whether to transitioncontrol back to the driver and send the vehicle into a safe stop modeafter the sixth incident of the driver being labeled as temporarilyunaware by the attention monitor 102 at fifteen minutes into the 15minute time window. A circle 504 l is shown indicating where theattention of the driver is directed when the attention monitor 102labeled the driver as temporarily unaware. In an example, the driverreceives a phone call and the attention of the driver moves from theforward roadway to the phone. The HMI 96 alerts the driver to payattention to the road. The HMIM 104 increments the temporarily unawarecriteria one incident to six incidents total.

Referring to FIG. 13 , a diagram is shown illustrating the HMIM 104after capturing the sixth incident of the driver being temporarilyunaware within the 15 minute time window. A picture 500 m is shownillustrating the view within the cockpit of the vehicle from theperspective of the driver. A display 502 m illustrates a current statusfor the criteria used by the HMIM 104 to determine whether to transitioncontrol back to the driver and send the vehicle into a safe stop modefollowing the sixth incident of inattentiveness at twenty minutes. Acircle 504 m is shown indicating where the attention of the driver isdirected. In an example, the HMIM 104 transitions control back to thedriver and safely discontinues the ADAS feature in response to sixincidents of the driver being inattentive within the fifteen minuteperiod. The HMI 96 alerts the driver to take control of the vehicle. Thevehicle begins to decelerate to 5 kph.

In general, when the HMIM 104 determines that something is wrong, theHMIM 104 requests that the driver take over operation of the vehicle andsafely discontinues the automation feature. Because discontinuingactivity of the automation feature at high speed may be unsafe as well,particularly when the driver is known to not be aware, the HMIM 104 mayuse the HMI 96 to present a request for the driver to take overoperation of the vehicle. In an example, the HMIM 104 may facilitate asafe transition by slowly degrading operation of the automation featureto a safe state (e.g., decelerating to a safe speed and/or a safe stop)before completely discontinuing collaborative operation. However, otherstrategies for discontinuing collaborative operation may be implementedto meet design criteria of a particular situation or application.

Referring to FIG. 14 , a diagram illustrating an electronic controlmodule implementing an advanced driver-assistance systems (ADAS) featurecontrol system in accordance with an example embodiment of the inventionis shown. In an example, an apparatus 800 may implement an electroniccontrol unit or module (ECU). In an example, the electronic controlmodule (ECU) 800 may be implemented as a domain controller (DC). Inanother example, the ECU 800 may be implemented as an active safetydomain master (ASDM). In various embodiments, the ECU 800 may beconfigured to control activation of one or more features (or functions)of an ADAS component of a vehicle. In various embodiments, the driverattentiveness estimator 100 may be implemented within the ECU 800. In anexample, the ECU 800 may be connected to the vehicle platform 92, thedriver monitoring system (DMS) 94, the human machine interface (HMI) 96,an electronic bus 802, and map and sensors 804 of the vehicle. In anexample, the ECU 800 may be configured to (i) receive the signalsVEHICLE SPEED, DEACTIVATION REQUEST, ACTIVATION REQUEST, and DRIVER INFOfrom the vehicle systems and communicate the signals DECELERATIONREQUEST and DRIVER WARNING to the systems of the vehicle.

In an example, the ECU 800 may be connected to a block (or circuit) 802.The circuit 802 may implement an electronic bus. The electronic bus 802may be configured to transfer data between the ECU 800 and the vehicleplatform 92, the DMS 94, the HMI 96, and the map and sensors 804 (e.g.,the HD map receiver 260, the GNSS receiver 262, the forward lookingcamera (FLC) 250, the corner/side radar sensors 252 a-252 n, the forwardlooking radar (FLR) sensor 254, and the inertial measurement unit 264.In some embodiments, the electronic bus 802 may be implemented as avehicle Controller Area Network (CAN) bus. The electronic bus 802 may beimplemented as an electronic wired network and/or a wireless network.Generally, the electronic bus 802 may connect one or more components ofthe vehicle 50 to enable a sharing of information in the form of digitalsignals (e.g., a serial bus, an electronic bus connected by wiringand/or interfaces, a wireless interface, etc.).

The ECU 800 generally comprises a block (or circuit) 820, a block (orcircuit) 822, a block (or circuit) 824, a block (or circuit) 826, and ablock (or circuit) 828. The circuit 820 may implement a processor. Thecircuit 822 may implement a communication port. The circuit 824 mayimplement a filter. The circuit 826 may implement a clock. The circuit828 may implement a memory. Other blocks (not shown) may be implemented(e.g., I/O ports, power connectors, interfaces, etc.). The number and/ortypes of circuits implemented by the module 800 may be varied accordingto the design criteria of a particular implementation.

The processor 820 may be implemented as a microcontroller, amulti-thread microprocessor, or any combination thereof. The processor820 may comprise a block (or circuit) implementing the attention monitor102, a block (or circuit) implementing the human machine interactionmonitor 104, and/or a block (or circuit) implementing the mode manage106. The processor 820 may comprise other components (not shown). Insome embodiments, the processor 820 may be a combined (e.g., integrated)chipset implementing processing functionality. In some embodiments, theprocessor 820 may be comprised of a number of separate circuits (e.g.,the microcontroller, the multi-thread microprocessor, a digital signalprocessor (DSP), a graphics processing unit (GPU), etc.). The design ofthe processor 820 and/or the functionality of various components of theprocessor 820 may be varied according to the design criteria of aparticular implementation. The processor 820 is shown sending data toand/or receiving data from the vehicle platform 92, the communicationport 822, and/or the memory 828.

The memory 828 may comprise a block (or circuit) 860 and a block (orcircuit) 862. The block 860 may store driver awareness (orattentiveness) estimator (DAE) data. The block 862 may store computerreadable instructions (e.g., instructions readable by the processor820). The DAE data 860 may store various data sets 870 a-870 n. Forexample, the data sets 870 a-870 n may comprise a count of transitionsto the temporarily unaware state 870 a, c a count of transitions to theunaware state 870 b, a count of transitions to the out-of-loop state 870c, a long-term glance distribution 870 d, driver info 870 e and/or otherdata 870 n.

In an example, the other data 870 n may comprise parameters (e.g.,coefficients) used to transform data received from the sensors (e.g.,FLC, FLR, FCR, FCS, and IMU). The calibration data 870 n may providemany sets of coefficients (e.g., one set of coefficients for each of thesensors). The calibration data 870 n may be updatable. For example, thecalibration data 870 n may store current values as coefficients for thesensors and as the data from the sensors drifts the module 800 mayupdate the calibration data 870 n in order to maintain accuracy. Theformat of the calibration data 870 n may vary based on the designcriteria of a particular implementation.

Various other types of data (e.g., the other data 870 n) may be storedas part of the DAE data 860. For example, the other data 970 n may storeglance distributions for a plurality of drivers. For example, the otherdata 870 n may store past data values of the calibration data and/orcurrent data values of the calibration data. The past and current datavalues of the calibration data may be compared to determine trends usedto extrapolate and/or predict potential future values for thecalibration data.

The processor 820 may be configured to execute stored computer readableinstructions (e.g., the instructions 862 stored in the memory 828). Theprocessor 820 may perform one or more steps based on the storedinstructions 862. In an example, steps of the instructions 862 may beexecuted/performed by the processor 820 and may implement one or more ofthe attention monitor 102, the human machine interaction monitor 104,and the mode manager 106. The instructions executed and/or the order ofthe instructions 862 performed by the processor 820 may be variedaccording to the design criteria of a particular implementation.

The communication port 822 may allow the module 800 to communicate withexternal devices such as the map and sensors 804, the vehicle platform92, the driver monitoring system 94, and the human machine interface 96.For example, the module 800 is shown connected to the externalelectronic bus 802. In an example, information from the module 800 maybe communicated to an infotainment device for display to a driver. Inanother example, a wireless connection (e.g., Wi-Fi, Bluetooth,cellular, etc.) to a portable computing device (e.g., a smartphone, atablet computer, a notebook computer, a smart watch, etc.) may allowinformation from the module 800 to be displayed to a user.

The filter 826 may be configured to perform a linear quadraticestimation. For example, the filter 824 may implement a Kalman filter.Generally, the filter 824 may operate recursively on input data toproduce a statistically optimal estimate. For example, the filter 824may be used to calculate the position coordinates 870 a and/or estimatethe accuracy of the position coordinates 870 a. In some embodiments, thefilter 824 may be implemented as a separate module. In some embodiments,the filter 824 may be implemented as part of the memory 828 (e.g., thestored instructions 862). The implementation of the filter 824 may bevaried according to the design criteria of a particular implementation.

The clock 826 may be configured to determine and/or track a time. Thetime determined by the clock 826 may be stored as the time stamp data870 c. In some embodiments, the clock 826 may be configured to comparetime stamps received from a GNSS receiver.

The module 800 may be configured as a chipset, a system on chip (SoC)and/or a discrete device. For example, the module 800 may be implementedas an electronic control unit (ECU). In some embodiments, the module 800may be configured to control activation of one or more ADASfeatures/functions.

Given the lack of state-of-the-art ASIL on HMI warning messages, anunderlying objective of the HMIM 104 in accordance with an embodiment ofthe invention is to provide a monitoring functionality that ensures thesufficient controllability of a supervising driver to possible hazardousevents. In an example, the HMIM 104 may achieve sufficientcontrollability by ensuring driver engagement. In an example, the HMIM104 may ensure driver engagement by monitoring glance distributionpatterns as a measure of attentiveness. In various embodiments, a fewexample functionality iterations of the HMIM 104 may be implemented tomitigate false positives during decision making.

In one example, the HMIM 104 may check the delta change in eye glanceshift pre and post HMI warnings to determine whether the warning isbeing conveyed to the driver and to safeguard against omission of HMImessages. For example, the HMIM 104 may subscribe to a signal indicating“eyes on road” in real time from a camera of the driver monitoringsystem 94. The signal indicating “eyes on road” may be used as feedbackto quickly evaluate whether there is improvement in glance distributionafter each escalation warning. If not, the HMIM 104 may fail safeappropriately.

In another example, the HMIM 104 may intentionally try to take the eyesof the driver off the road momentarily when the attention monitor 102reports the driver being in the “aware” state for a longer duration thanexpected to safeguard against false positive from the camera of the DMS94 and attention monitor attention levels. When the driver has beenreported to be “aware” for the longer time duration, the HMIM 104 maysend directed prompts to divert attention of the driver away from road(when judged to be safe to do so by subscribing to environmentalinformation from onboard sensors, GNSS, HD map, etc.) and verify whetherthe front-end signal chain (e.g., DMS 94, attention monitor 102, etc.)detect the diverted attention. If not, the HMIM 104 may fail safeappropriately.

In another example, other cabin sensory information may be integratedinto the HMIM 104 as inputs to form a holistic driver state estimation.In addition to eyes on road information, the HMIM 104 may subscribe tohands on steering wheel, pedal information, seat sensors, seat beltstatus, etc. to form a holistic driver state estimation model. The HMIM104 may leverage feedback from each of these inputs to detect andmitigate inattentiveness.

In still another example, the HMIM 104 may be developed with artificialintelligence/machine learning (AI/ML) based non-deterministic algorithmsto baseline a driver attentiveness profile for each individual driverand track inattentiveness against the baseline of the particular driver.In various embodiments, each vehicle may implement a generic HMIM 104that over time may customize itself by tracking and learning about aninattentiveness profile of the driver by baselining the inattentivenessprofile of the driver against the glance distribution of the same driverduring manual driving. Inattentiveness during supervised driving maythen be flagged when the inattentiveness exceeds the threshold notedduring prior manual driving.

The terms “may” and “generally” when used herein in conjunction with“is(are)” and verbs are meant to communicate the intention that thedescription is exemplary and believed to be broad enough to encompassboth the specific examples presented in the disclosure as well asalternative examples that could be derived based on the disclosure. Theterms “may” and “generally” as used herein should not be construed tonecessarily imply the desirability or possibility of omitting acorresponding element.

The designations of various components, modules and/or circuits as“a”-“n”, when used herein, disclose either a singular component, moduleand/or circuit or a plurality of such components, modules and/orcircuits, with the “n” designation applied to mean any particularinteger number. Different components, modules and/or circuits that eachhave instances (or occurrences) with designations of “a”-“n” mayindicate that the different components, modules and/or circuits may havea matching number of instances or a different number of instances. Theinstance designated “a” may represent a first of a plurality ofinstances and the instance “n” may refer to a last of a plurality ofinstances, while not implying a particular number of instances.

While the invention has been particularly shown and described withreference to embodiments thereof, it will be understood by those skilledin the art that various changes in form and details may be made withoutdeparting from the scope of the invention.

1. An apparatus comprising: an interface configured to receive aplurality of sensor signals from a vehicle platform of a vehicle andpresent one or more control signals to the vehicle platform; and acontrol circuit configured to (i) detect whether an attention state of adriver is in an attentive state or an inattentive state in response toone or more of the plurality of sensor signals from the vehicle platformduring a first window having a first duration, (ii) assess whether thedriver is sufficiently attentive by monitoring the one or more of theplurality of sensor signals from the vehicle platform and determiningwhether changes in the attention state of the driver during a secondwindow having a second duration that is longer than the first durationexceeds a threshold, and (iii) when the threshold is exceeded,transition operation of the vehicle to the driver and safely discontinuean automation system function of the vehicle.
 2. The apparatus accordingto claim 1, wherein said control circuit comprises a driver attentionestimator configured to (i) generate a first control signalcommunicating the attention state of the driver during the first windowhaving the first duration and (ii) generate a second control signalcommunicating an assessment of whether the driver is sufficientlyattentive for the automation system function to safely continue tooperate the vehicle.
 3. The apparatus according to claim 2, wherein saiddriver attention estimator comprises: an attention monitor configured todetermine the attention state of the driver during the first windowhaving the first duration and generate the first control signal; and ahuman machine interaction monitor configured to generate the assessmentof whether the driver is sufficiently attentive during the second windowhaving the second duration and generate the second control signal. 4.The apparatus according to claim 3, wherein said human machineinteraction monitor is further configured to assess whether a humanmachine interface of the vehicle is successfully communicating with thedriver.
 5. The apparatus according to claim 3, wherein said humanmachine interaction monitor is further configured to determine whetherthe driver is gaming the attention monitor.
 6. The apparatus accordingto claim 3, wherein said human machine interaction monitor is furtherconfigured to monitor a plurality of driver awareness states of theattention monitor.
 7. The apparatus according to claim 6, wherein saidplurality of driver awareness states of the attention monitor compriseone or more of an aware state, a temporarily unaware state, and unawarestate, and an out-of-the-loop state.
 8. The apparatus according to claim6, wherein said human machine interaction monitor is further configuredto use a respective threshold for each of the plurality of driverawareness states of the attention monitor.
 9. The apparatus according toclaim 8, wherein the respective threshold for each of the plurality ofdriver awareness states of the attention monitor are programmable. 10.The apparatus according to claim 8, wherein the respective threshold foreach of the plurality of driver awareness states of the attentionmonitor comprises a maximum acceptable number of transitions from theaware state to a corresponding driver awareness state during the secondwindow having the second duration.
 11. The apparatus according to claim8, wherein the human machine interaction monitor is further configuredto customize itself over time by tracking and learning aninattentiveness profile of the driver.
 12. The apparatus according toclaim 11, wherein the human machine interaction monitor is furtherconfigured to learn the inattentiveness profile of the driver bybaselining the inattentiveness profile of the driver against a glancedistribution of the driver during manual driving.
 13. The apparatusaccording to claim 3, wherein said control circuit further comprises afeature mode manager configured to activate or maintain operation of theautomation system of the vehicle when the threshold is not exceeded andsafely transfer operation of the vehicle from the automation system tothe driver when the threshold is exceeded.
 14. The apparatus accordingto claim 13, wherein said feature mode manager is configured to send awarning to the driver to take over control of the vehicle via a humanmachine interface of the vehicle and degrade performance of an autopilotfunction of the automation system by generating a deceleration requestto the vehicle platform.
 15. A method of controlling an automationsystem function of a vehicle comprising: receiving a plurality of sensorsignals from a vehicle platform of a vehicle; detecting whether anattention state of a driver is in an attentive state or an inattentivestate in response to one or more of the plurality of sensor signals fromthe vehicle platform during a first window having a first duration;assessing whether the driver is sufficiently attentive by monitoring theone or more of the plurality of sensor signals from the vehicle platformand determining whether changes in the attention state of the driverduring a second window having a second duration that is longer than thefirst duration exceeds a threshold; and when the threshold is exceeded,transferring operation of the vehicle to the driver and safelydiscontinuing the automation system function of the vehicle.
 16. Themethod according to claim 15, further comprising using a driverattention estimator of an electronic control unit of the vehicle to (i)generate a first control signal communicating the attention state of thedriver during the first window having the first duration and (ii)generate a second control signal communicating an assessment of whetherthe driver is sufficiently attentive for the automation system functionto safely continue to operate the vehicle.
 17. The method according toclaim 16, wherein said driver attention estimator comprises: anattention monitor configured to determine the attention state of thedriver during the first window having the first duration and generatethe first control signal; and a human machine interaction monitorconfigured to generate the assessment of whether the driver issufficiently attentive during the second window having the secondduration and generate the second control signal.
 18. The methodaccording to claim 17, further comprising using the human machineinteraction monitor to assess whether a human machine interface of thevehicle is successfully communicating with the driver.
 19. The methodaccording to claim 17, wherein the human machine interaction monitor isfurther configured to: monitor a plurality of driver awareness states ofthe attention monitor; and use a respective threshold for each of theplurality of driver awareness states of the attention monitor.
 20. Themethod according to claim 19, wherein the respective threshold for eachof the plurality of driver awareness states of the attention monitorcomprises a maximum acceptable number of transitions from an aware stateto a corresponding driver awareness state during the second windowhaving the second duration.